Position based energy minimizing function

ABSTRACT

A computer system is provided that includes an input system and a processor that may be configured to control a virtual manipulator based on input data received from the input system. The processor is further configured to determine an initial state of a system that includes at least an initial state of the virtual manipulator and an initial state of a virtual object, detect at least one contact point between a portion of the virtual manipulator and the virtual object. The processor is further configured to calculate a subsequent state for the virtual object that minimizes a set of energies or residuals defined in terms of the one or more positional quantities determined for the initial state of the system and the one or more positional quantities determined for the subsequent state of the system using a position-based energy minimizing function.

BACKGROUND

Mixed reality three-dimensional interfaces may incorporate hand trackingto manipulate virtual objects. These interfaces may identify gestureinputs, such as an air tap, to begin interaction between the user's handand those virtual objects.

SUMMARY

A computer system is provided that comprises an input system and aprocessor configured to control a virtual manipulator based on inputdata received from the input system. The virtual manipulator isconfigured to manipulate a virtual object in a virtual environment. Theprocessor is further configured to determine an initial state of asystem that includes at least an initial position of the virtualmanipulator and an initial position of the virtual object, detect atleast one contact point between a portion of the virtual manipulator andthe virtual object. The processor is configured to determine one or morepositional quantities for the initial state of the system, detect asubsequent position of the virtual manipulator, and determine one ormore positional quantities for a subsequent state of the system thatincludes the subsequent state of the virtual manipulator. The processoris further configured to calculate a subsequent state for the virtualobject that minimizes a set of energies or residuals defined in terms ofthe one or more positional quantities determined for the initial stateof the system and the one or more positional quantities determined forthe subsequent state of the system using a position-based energyminimizing function. The processor is further configured to move thevirtual object to the calculated subsequent state.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic view of an example computer system thatperforms physics simulations using position based energy minimizingfunctions.

FIG. 2 shows an example head mounted display device configuration of theexample computer system of FIG. 1.

FIG. 3 shows an example of hand tracking to control virtual manipulatorsusing the computer system of FIG. 1.

FIG. 4 shows a flowchart for an example method of manipulating virtualobjects with virtual manipulators using position based energy minimizingfunctions.

FIG. 5A shows a virtual manipulator contacting a virtual object in avirtual environment executed by the computer system of FIG. 1.

FIG. 5B shows the virtual manipulator of manipulating a virtual objectin the virtual environment executed by the computer system of FIG. 1.

FIG. 6 shows a flowchart for an example method of manipulating a virtualobject by attaching the virtual object to a frame of reference of avirtual manipulator in the virtual environment executed by the computersystem of FIG. 1.

FIG. 7A shows an example virtual manipulator grasping a virtual objectwith two articulable portions in the virtual environment executed by thecomputer system of FIG. 1.

FIG. 7B shows an example virtual manipulator grasping and moving avirtual object with two articulable portions in the virtual environmentexecuted by the computer system of FIG. 1.

FIG. 8 shows a flowchart for a method of calculating positions andnormals for contact points between a virtual object and a virtualmanipulator in the virtual environment executed by the computer systemof FIG. 1.

FIG. 9A shows an example line that extends between and outward from twoclosest points on a virtual manipulator in the virtual environmentexecuted by the computer system of FIG. 1.

FIG. 9B shows an example of detecting contact points based on anintersection between the line and a surface of the virtual object in thevirtual environment executed by the computer system of FIG. 1.

FIG. 10A show an example of a simplified geometric representation of avirtual object in the virtual environment executed by the computersystem of FIG. 1.

FIG. 10B shows an example of determining one or more positionalquantities for an initial state of a system in the virtual environmentexecuted by the computer system of FIG. 1.

FIG. 11 shows an example of a set of energies or residuals resultingfrom moving the virtual manipulator of FIG. 10B to a subsequent state inthe virtual environment executed of the computer system of FIG. 1.

FIG. 12 shows a flowchart of a method for manipulating a virtual objectusing a plurality of predetermined grasping points on a virtualmanipulator in the virtual environment executed by the computer systemof FIG. 1.

FIG. 13A shows an example of the plurality of grasping points fixed on avirtual manipulator in the virtual environment executed by the computersystem of FIG. 1.

FIG. 13B shows an example coordinate system transformation for theplurality of grasping points in the virtual environment executed by thecomputer system of FIG. 1.

FIG. 14 shows a flowchart of a method for manipulating a virtual objectusing at least two virtual manipulators in the virtual environmentexecuted by the computer system of FIG. 1.

FIG. 15A shows an example of a virtual object being grasped by twovirtual manipulators in the virtual environment executed by the computersystem of FIG. 1.

FIG. 15B shows an example of a virtual object being manipulated by twovirtual manipulators in the virtual environment executed by the computersystem of FIG. 1.

FIG. 16 shows a flowchart of a method for scaling a virtual object usingtwo virtual manipulators in the virtual environment executed by thecomputer system of FIG. 1.

FIG. 17A shows a virtual object being scaled using two virtualmanipulators in the virtual environment executed by the computer systemof FIG. 1.

FIG. 17B shows a virtual object being scaled using two virtualmanipulators in the virtual environment executed by the computer systemof FIG. 1.

FIG. 18 shows a schematic view of an example computing environment inwhich the computer system of FIG. 1 may be enacted.

DETAILED DESCRIPTION

Mixed reality three-dimensional interfaces may incorporate hand trackingto manipulate virtual objects. These interfaces may identify gestureinputs, such as an air tap, to begin interaction between the user's handand those virtual objects. Typically, these mixed reality interfaces useimpulse based physics to simulate dynamics at the contact points betweenthe user's hands and the virtual objects. These impulse based physicssimulations typically treat each object as a rigid body, and attempts tosimulate the forces that would be applied to those objects using impulseforce equations. However, the act of grasping objects includes complexdynamics that typical real-time interaction systems incorrectly model.For example, grasping objects may include factors such as softdeformation of the hand, specific friction properties of the hand andthe virtual object, and other factors that typical system incorrectlysimulate.

These impulse based physics simulators may potentially improperlysimulate grasping physics by incorrectly calculating forces and torquesdespite the user intending to hold an object still, thus causing theobject to accelerate in unintended ways. These simulations may alsopotentially include large potential energies and instability due to deeppenetrations of the user's hand into the virtual objects.

To address these issues, FIG. 1 illustrates a computer system 10 that isconfigured to perform physics simulations using position-based energyminimizing functions. Computer system 10 may take the form of a headmounted display (HMD) device, a desktop computer device, a mobilecomputer device, or another suitable form. The computer system 10comprises a processor 12, a non-volatile memory device 14, a volatilememory device 16, an input system 18, and a display device 20. In oneexample, the input system 18 may include a camera device 22 and/oranother input device 24. It should be appreciated that the computersystem 10 may include other computer components not specificallyenumerated herein.

The camera device 22 may include a red-green-blue (RGB) camera and adepth camera configured to take RGB and depth images of a scene in frontof the camera device 22. In one example, the camera device 22 mayinclude one or more cameras located in different positions in thephysical environment. In an HMD device example, the camera device 22 maytake the form of outward facing cameras on the HMD device.

In another example, the input system 18 may include one or more othertypes of input devices 24, such as, for example, a keyboard and mouse, ahandheld controller, accelerometer and inertial sensor devices on an HMDdevice, etc. In one example, the display device 20 may take the form ofa stand-alone display, a display screen of a mobile device, a largeformat display, etc. In an HMD device example, the display device 20 maytake the form of a near-eye display integrated with the HMD device.

FIG. 2 illustrates an example computer system 10 in the form of an HMDdevice 26. The HMD device 26 may be worn by a user according to anexample of the present disclosure. The HMD device 26 may include a frame28 that wraps around the head of the user to position the display device20, which takes the form of a near-eye display in this example, close tothe user's eyes. In one example, the near-eye display 20 may take theform of a substantially non-see-through display in a virtual reality HMDdevice configuration. In other examples, the near-eye display 20 maytake the form of an at least partially see-through display that issupported in front of a viewer's eye or eyes in an augmented reality HMDdevice configuration.

As illustrated in FIG. 2, the frame supports additional components ofthe HMD device 26, such as, for example, the processor 12 and the inputsystem 18. The processor 12 includes logic and associated computermemory configured to provide image signals to the display device 20, toreceive sensory signals from the input system 18, and to enact variouscontrol processes described herein.

Any suitable display technology and configuration may be used to displayimages via the display device 20. For example, in a non-augmentedreality configuration, the display device 20 may be a non-see-throughLight-Emitting Diode (LED) display, a Liquid Crystal Display (LCD), orany other suitable type of non-see-through display. In an augmentedreality configuration, the display device 20 may be configured to enablea wearer of the HMD device 26 to view a physical, real-world object inthe physical environment through one or more partially transparentpixels displaying virtual object representations. For example, thedisplay device 20 may include image-producing elements such as, forexample, a see-through Organic Light-Emitting Diode (OLED) display.

As another example, the HMD device 26 may include a light modulator onan edge of the display device 20. In this example, the display device 20may serve as a light guide for delivering light from the light modulatorto the eyes of a wearer. In other examples, the display device 20 mayutilize a liquid crystal on silicon (LCOS) display.

The input system 18 may include various sensors and related systems toprovide information to the processor 12. Such sensors may include aninertial measurement unit (IMU) 24A. The camera device 22 may includeone or more outward facing camera devices 22A. In one example, thecamera device 22 may also include one or more inward facing cameradevices 22B. The one or more inward facing camera devices 22B may beconfigured to acquire image data in the form of gaze tracking data froma wearer's eyes.

The one or more outward facing camera devices 22A may be configured tocapture images of a scene in front of the HMD device 26. In one example,the one or more outward facing camera devices 22A may include avisible-light camera or RBG camera configured to collect a visible-lightimage of a physical space. Further, the one or more outward facingcamera devices 22A may include a depth camera configured to collect adepth image of a physical space. More particularly, in one example thedepth camera is an infrared time-of-flight depth camera. In anotherexample, the depth camera is an infrared structured light depth camera.

Data from the outward facing camera devices 22A may be used by theprocessor 12 to generate and/or update a three-dimensional (3D) model ofthe physical environment. Data from the outward facing camera devices22B may be used by the processor 12 to identify and track a hand of auser that is in front of the HMD device 26. The processor 12 may executeinstructions to track the user's hand and control a virtual manipulatorbased on the user's tracked hand, as will be described in more detailbelow.

In augmented reality configurations of HMD device 26, the positionand/or orientation of the HMD device 26 relative to the physicalenvironment may be assessed so that augmented-reality images may beaccurately displayed in desired real-world locations with desiredorientations. For example, processor 12 may execute instructions togenerate a 3D model of the physical environment and perform simultaneouslocalization and mapping processes. In both augmented reality andnon-augmented reality configurations of HMD device 26, the IMU 24A ofHMD device 26 may be configured to provide position and/or orientationdata of the HMD device 26 to the processor 12. In one implementation,the IMU 24A may be configured as a three-axis or three-degree of freedom(3DOF) position sensor system. This example position sensor system may,for example, include three gyroscopes to indicate or measure a change inorientation of the HMD device 26 within 3D space about three orthogonalaxes (e.g., roll, pitch, and yaw). The orientation derived from thesensor signals of the IMU may be used to display, via the display device20, one or more holographic images with a realistic and stable positionand orientation.

In another example, the IMU 24A may be configured as a six-axis orsix-degree of freedom (6DOF) position sensor system. Such aconfiguration may include three accelerometers and three gyroscopes toindicate or measure a change in location of the HMD device 26 alongthree orthogonal spatial axes (e.g., x, y, and z) and a change in deviceorientation about three orthogonal rotation axes (e.g., yaw, pitch, androll). In some implementations, position and orientation data from theoutward facing camera devices 22A and the IMU 24A may be used inconjunction to determine a position and orientation (or 6DOF pose) ofthe HMD device 26.

In some examples, a 6DOF position sensor system may be used to displayholographic representations in a world-locked manner. A world-lockedholographic representation appears to be fixed relative to one or morereal world objects viewable through the HMD device 24, thereby enablinga wearer of the HMD device 24 to move around a real world physicalenvironment while perceiving a world-locked hologram as remainingstationary in a fixed location and orientation relative to the one ormore real world objects in the physical environment.

Images captured by the one or more outward facing cameras 22A may beused to track the hands of a user and control one or more virtualmanipulators. FIG. 3 shows an example of a user's hands 30 in the fieldof view 32 of the one or more outward facing cameras 22A. The processor12 may be configured to perform skeletal tracking of the user's hands 30to track a position of the user's hands 30. The processor 12 may befurther configured to track the positions and orientations of one ormore phalanges of the user's hands 30. As illustrated in FIG. 3, theprocessor 12 may be configured to control the position, orientation, andmovement of one or more virtual manipulators 34 based on the tracking ofthe user's hands 30. In one example, the one or more virtualmanipulators 34 may include virtual phalanges that are also controlledbased on the tracking of the user's hands 30. In other examples, the oneor more virtual manipulators 34 may be controlled based on other typesof input to the input system 18, such as, for example, controller inputto a handheld controller, or input to a keyboard and mouse.

Turning back to FIG. 1, the processor 12 is configured to execute avirtual environment 38 that may take the form of a three-dimensionalvirtual scene and may include the one or more virtual manipulators 34and one or more virtual objects 40. The virtual environment 38 mayinclude a position-based physics engine 42 that calculates physicalinteractions between the one or more virtual manipulators 34 and the oneor more virtual objects 40 based on position based energy minimizingfunctions 44 that will be described in more detail below.

The virtual environment 38 may include a virtual manipulator controller46 that is configured to control the one or more virtual manipulators 34based on input data 36 received from the input system 18 of the computersystem 10. As discussed above, in one example, the input data 36 mayinclude image data from the camera device 22 that captures images, suchas RGB and depth images, of a user's hands. The virtual manipulatorcontroller 46 may be configured to track a position of a user's hand viathe input data 36, such as image data, received from the input system18. The virtual manipulator controller 46 may be further configured totrack the three-dimensional positions and orientations of one or morephalanges of the user's hands and determined a three-dimensional poseestimate for the user's hand. It should be appreciated that the virtualmanipulator controller 46 may utilize any suitable type of humanskeletal tracking processes and methods to track positions andorientations of the user's hands, phalanges of the user's hands, andother skeletal structures of the user's hands based on the image datacaptured by the camera device 22 of the input system 18.

As illustrated in FIG. 3, the one or more virtual manipulators 34 maytake the form of a graphical representation of a hand. However, itshould be appreciated that the one or more virtual manipulators 34 maytake other suitable forms, such as, for example, grasping tools, claws,robotic arms, etc. The virtual manipulator controller 36 may beconfigured to control the graphical representation of the hand based onthe tracked position of the user's hand.

In another example, the virtual manipulator controller 46 may beconfigured to control the one or more virtual manipulators 34 based onother types of input data 36 received by other types of input devices 24of the input system 18. For example, the one or more virtualmanipulators 34 may be controlled based on input data 36 to a handheldcontroller. As another example, the input data 36 may be received by akeyboard and mouse input device. However, it should be appreciated thatthe virtual manipulators 34 may be controlled via any suitable type ofinput data 36 and input device 24.

As illustrated in FIG. 1, the virtual manipulator controller 36 maycontrol and move the one or more virtual manipulators 34 between variousstates 48 in the three-dimensional virtual environment 38. As usedherein, the term “state” may be used to refer to a three-dimensionalpose of the virtual manipulator and/or virtual object within thethree-dimensional virtual environment. The three-dimensional pose may bedetermined with six degrees of freedom (6DoF), and may include therelative position and orientations of one or more parts/portions of thevirtual manipulator and virtual object relative to one another and/or acoordinate system defined for the three-dimensional virtual environment.In one example, the virtual manipulators 34 may include one or morearticulable portions 50 that may be controlled based on the input data36. In hand tracking embodiments, the articulable portions 50 of thevirtual manipulators 34 may be controlled based on the positions ofcorresponding phalanges of the user's hands tracked based on the imagedata received by the camera device 22. The virtual manipulatorcontroller 46 may control the virtual manipulators 34 to move thevirtual manipulators from an initial state 48A through subsequent states48B, as will be discussed in more detail below.

The one or more virtual manipulators 34 may be configured to manipulatethe one or more virtual objects 40 in the virtual environment 38according to rules and logic of the physics engine 42 of the virtualenvironment 38. For example, the one or more virtual manipulators 34 maybe configured to push, pull, grasp, move, rotate, resize, and otherwisemanipulate the one or more virtual objects 40 in the virtual environment38 as mediated by the physics engine 42. For example, the virtualmanipulators 34 may be configured to grasp the virtual objects 40 andmove the virtual objects 40 from an initial state of the virtual object52A to a subsequent state of the virtual object 52B in the virtualenvironment 38.

The physics engine 42 may include a collision detection module 54configured to detect contacts between the virtual manipulator 34 andsurfaces of the one or more virtual objects 40. In one example, acontact between a portion of a virtual manipulator and a surface of avirtual object 40 may be detected based on determining that the portionof the virtual manipulator penetrates the surface of the virtual object.In another example, the contact may be determined based on detectingthat the portion of the virtual manipulator, such as, for example, acontact point on the virtual manipulator, is within a threshold distanceof a surface of the virtual object. It should be appreciated that othertypes of collision detection processes and techniques may be used by thecollision detection module 54 to detect contacts between virtualmanipulators and virtual objects.

The virtual environment 38 including the virtual manipulators 34 and thevirtual objects 40 may be displayed via the display device 20. As thevirtual manipulators 34 and virtual objects 40 are moved from an initialsystem state to a subsequent system state in the virtual environment 38,updated frames for the virtual environment 38 may be rendered anddisplayed to the user via the display device 20. In augmented realityHMD device configurations, the virtual environment 38 may be displayedon top of the real world environment via a see-through near-eye display.

FIG. 4 illustrates an example computer-implemented method 400 forcalculating a subsequent state 52B for a virtual object 40 according toa position based energy minimizing functions 44 of the physics engine42. At 402, the method 400 may include determining an initial state of asystem 41 that includes at least an initial state 48A of the virtualmanipulator 34 and an initial state 52A of the virtual object 40. Asillustrated with reference to FIG. 5A, the virtual manipulator 34 has aninitial state 48A, and the virtual object 40 has an initial state 52A.As used herein, term “system” may be used to refer to a set of states ofthe virtual manipulators 34, virtual objects 40, and contact pointsbetween the virtual manipulators and virtual objects at a particularpoint in time of the virtual environment, as illustrated in FIG. 1.

At 404, the method 400 may include detecting at least one contact point56 between a portion 58 of the virtual manipulator 34 and the virtualobject 40. As illustrated with reference to FIG. 5A, the collisiondetection module 54 detects contact between the portion 58 of thevirtual manipulator 34 and a surface of the virtual object 40. In theillustrated example, the portion 58 is a pointer finger of the virtualmanipulator taking the form of a graphical representation of a hand.However, it should be appreciated that the portion 58 may take otherforms, such as, for example, two or more phalanges of the graphicalrepresentation of the hand, claws on a robot representation of thevirtual manipulator, etc. As discussed above, the contact 56 may bedetected via a variety of collision and penetration techniques, such as,for example, detecting the contact point 56 based on detecting apenetration of the portion 58 of the virtual manipulator 34 into asurface of the virtual object 40.

At 406, the method 400 may include determining one or more positionalquantities for the initial state of the system 41. The one or morepositional quantities may be determined for one or more of the virtualmanipulator 34, the virtual object 50, and the at least one contactpoint 56 between the portion 58 of the virtual manipulator 34 and thevirtual object 50 at the initial state of the system 41. In one example,the one or more positional quantities may include a frame of referenceof the virtual manipulator, a frame of reference of the virtual object,a scaling frame of reference, a position of the contact point, a normalof the contact point, a penetration depth of the contact point, andother suitable types of positional quantities. In one example, thepositional quantities may include a contact point tangent component forfriction, which will be discussed in more detail below.

As illustrated with reference to FIG. 5A, the one or more positionalquantities may include a position P1 of the contact point 56 relative tothe portion 58 of the virtual manipulator 34 and the virtual object 50that may be calculated by the processor. In one example, the position P1may indicate a degree of penetration of a point on the portion 58 of thevirtual manipulator 34 into the geometry of the virtual object 40. Thephysics engine 42 may also calculate a normal 60 for the contact point56 between the portion 58 of the virtual manipulator 34 and the virtualobject 50 as one of the positional quantities. In one example, thenormal 60 may be calculated based on a geometry of the surface of thevirtual object 40 at the point of contact. In another example, thenormal 60 may be calculated based on an idealized geometricrepresentation of the virtual object 40, such as, for example, a sphere,that is positioned at the point of contact 56. Other techniques andmethods for calculating the normal 60 will be described herein.

At 408, the method 400 may include detecting a subsequent state 48B ofthe virtual manipulator 34. As illustrated with reference to FIG. 5B,new input data has been received that moves the virtual manipulator 34from the initial state 48A to the subsequent state 48B. The processor 12may then be configured to compute a subsequent state of the system 41that includes the subsequent state of the virtual manipulator 48B.

In typical physics engines, in order to calculate how the movement ofthe virtual manipulator 34 would affect the virtual object 40, thesetypical physics engines would calculate a force or impulse applied bythe virtual manipulator to the virtual object 40 over a period of time.These typical physics engines would also need to calculate otherpotential forces, such as, for example, friction between the virtualmanipulator and virtual object, mass of the virtual object, etc.However, due to the complexity of these types of force calculations,simulation errors and processing resource consumptions required tosimulate the impulse based physics may potentially increase, asdiscussed above.

Method 400 uses energy minimizing position based dynamics techniquesdescribed herein to calculate how the virtual manipulator 34 will affectthe virtual object 40. At 410, the method 400 may include determiningone or more positional quantities for a subsequent state of the system41 that includes the subsequent state 48B of the virtual manipulator 34.Step 410 may include updating and/or transforming one or more positionalquantities of the system 41, such as, for example, the position andnormal 60 for the at least one contact point 56 based on the subsequentstate of the virtual manipulator 48B in the subsequent state of thesystem 41. As illustrated with reference to FIG. 5B, the position andnormal 60 for the contact point 56 have been transformed and/or updatedbased on the subsequent state of the system 41 including the subsequentstate 48B of the virtual manipulator 34. Specifically, in this example,the position and normal 60 have been translated a distance D1 based onthe difference between the initial and subsequent states of the virtualmanipulator 34.

At 412, the method 400 may include calculating a subsequent state 52Bfor the virtual object 40 that minimizes a set of energies or residualsdefined in terms of the one or more positional quantities determined forthe initial state of the system 41 and the one or more positionalquantities determined for the subsequent state of the system 41 using aposition-based energy minimizing function 44. As illustrated withreference to FIG. 5B, the position and normal of the contact point 56have been transformed/updated to the position P2 and normal 60T. In thisspecific example, the virtual manipulator 34 has only been translated adistance D1 to the right along a linear path. Thus, whentransforming/updating the position and normal of the contact point 56,the position was also translated the distance D1. The orientation of thevirtual manipulator 34 did not change from the initial to subsequentposition, thus the orientation of the normal 60 also did not change.Thus, the processor 12 may calculate the subsequent state 52B of thevirtual object 40 to minimize the set of energies or residuals, such as,for example, the residual for the distance D1 that indicates thedisparity between the position P1 and the transformed position P2 forthe contact point 56.

It should be appreciated that the example set of energies or residualsshown in FIGS. 5A and 5B are merely illustrative, and that the positionbased energy minimizing function may be configured to minimize othertypes of energies and residuals defined in terms of other types ofpositional quantities. For example, the minimized energies or residualsmay include a kinetic energy defined by a difference of virtual objectpositions between the initial and subsequent states of the system 41. Asanother example, the minimized energies or residuals may include apenetration depth residual defined by a calculated depth of apenetration of the portion of the virtual manipulator 34 into thevirtual object 40 at the subsequent state of the system 41. As yetanother example, the minimized energies or residuals may include atangential friction residual defined by a difference between twopositional quantities at the subsequent state of the system 41 (e.g. aposition of the transformed contact point on the virtual manipulatorgiven a manipulator transform from the initial state to the subsequentstate, minus a position of the transformed contact point on the virtualobject given an object transform from the initial state to thesubsequent state). As yet another example, the minimized energies orresiduals may include an angular friction residual defined by adifference between two positional quantities at the subsequent state ofthe system 41. It should be appreciated that other types of energies andresiduals not specifically mentioned above may be minimized by theposition based energy minimizing functions 44.

At 414, the method 400 may include moving the virtual object 40 to thecalculated subsequent state 52B. The virtual manipulator 34 and thevirtual object 40 may then be displayed to the user via the display 20.It should be appreciated that the virtual object 40 was moved to thesubsequent state 52B based on the positional quantities of the initialand subsequent states of the system 41, and did not require computingimpulse and force calculations as in typical physics engines.

FIG. 6 illustrates an example computer implemented method 600 forcalculating the subsequent state of the virtual object 40 based on oneor more positional quantities that include a frame of reference of thevirtual manipulator. At 602, the method 600 may include, based ondetecting the at least one contact point 56 between the portion 58 ofthe virtual manipulator 34 and the virtual object 40, determining arelative state of the virtual object 40 compared to the frame ofreference of the virtual manipulator 34 centered at a reference point onthe virtual manipulator. As illustrated with reference to FIG. 7A, thevirtual manipulator 34 has an initial state 48A and the virtual object40 has an initial state 52A in the initial state of the system 41 of thevirtual environment. Based on these positions, the processor 12 detectstwo contact points 56A and 56B between two portions 58A and 58B of thevirtual manipulator 34 according to any of the techniques describedherein. Based on the detected contact points, the processor 12 maydetermine that the user is intending to grasp the virtual object 40. Theprocessor 12 may then be configured to determine a relative state of thevirtual object 40 to a frame of reference centered at a reference point64 on the virtual manipulator 34. In the illustrated example, therelative state of the virtual object 40 is indicated by the vector V1.In one example, the reference point 64 is a point on a palm of thevirtual manipulator 34. However, it should be appreciated that thereference point 64 may be set at other suitable points on the virtualmanipulator, such as a wrist, a finger, etc.

At 604, the method 600 may include calculating the subsequent state 52Bfor the virtual object 40 that minimizes an energy or residual definedin terms of the relative state (denoted by vector V1) of the virtualobject 40 compared to the frame of reference of the virtual manipulator34 for the initial state of the system 41 and the relative state of thevirtual object 40 compared to the frame of reference of the virtualmanipulator 34 for the subsequent state of the system 41. As illustratedwith reference to FIG. 7B, the virtual manipulator 34 has moved to asubsequent state 48B. The processor 12 may be configured to transformand/or update the relative state of the virtual object 40 based on thesubsequent state of the virtual manipulator 34. The transformed relativestate is illustrated as the vector V2 in FIG. 7B. Next, the processor 12may be configured to calculate a subsequent state 52B for the virtualobject 40 that minimizes a set of residuals, such as the angle A,defined in terms of a disparity between the initial relative state andthe transformed relative state.

In this manner, the virtual object 40 is attached to the reference point64 of the virtual manipulator 34 such that transformations applied tothe virtual manipulator 34 are also correspondingly applied to thevirtual object 40 so that the virtual object 40 appears to remaingrasped by the virtual manipulator 34 while moving through the virtualenvironment.

In order to ungrasp the virtual object and detach the virtual object 40from the virtual manipulator 34, the processor may be configured todetect that the one or more portions 58A and 58B are no longercontacting a surface of the virtual object 40. Upon detecting that theone or more portions 58A and 58B are no longer contacting the virtualobject 40, the processor may be configured to stop tracking the relativeposition of the virtual object 40 to the reference point 64 so thattransformation applied to the virtual manipulator are no longer alsoapplied to the virtual object 40.

In one example, the contact points 56 between the virtual manipulator 34and the virtual object 40 may be detected based on a point of overlapbetween the portion 58 of the virtual manipulator and the virtual object40. In this example, the positions and normals for these contact points56 may be calculated based on surface geometry of the portion 58 of thevirtual manipulator 34 and surface geometry of the virtual object 40.Other techniques for calculating the positions and normal for thesecontact points 56 are described herein.

In the example illustrated in FIG. 7A, the processor may be configuredto detect contact points 56 for each portion of the virtual manipulator34 that is contacting the virtual object. In examples where the virtualmanipulator 34 is a graphical representation of a hand, such as in FIG.7A, the first and second portions 58A and 58B may take the form of afirst articulable portion and a second articulable portion, such as afinger and a thumb. The processor 12 may be configured to detect atleast two contact points including a first contact point 56A between thefirst articulable portion 58A and the virtual object 40 and a secondcontact point 56B between the second articulable portion 58B and thevirtual object 40. The processor 12 may be configured to perform thetechniques described in method 400 and and/or method 600 using both thefirst contact point 56A and the second contact point 56B. In someexamples, the virtual manipulator may include more than two articulableportions. For example, the graphical representation of the hand mayinclude an articulable portion for each thumb and finger, and theprocessor 12 may be configured to detect contact points between each ofthese articulable portions and the virtual object 40.

It should be appreciated that other techniques may be used to calculatecontact points and normals at those contact points. For example, FIG. 8illustrates an example method 800 for calculating contact points andnormals for a virtual object being grasped between two articulableportions having opposing normals. As illustrated with reference to FIG.9A, the virtual manipulator includes at least the first articulableportion 58A and the second articulable portion 58B. In this example, thefirst articulable portion 58A is a thumb of a graphical representationof a hand, and the second articulable portion 58B is a pointer finger ofthe graphical representation of the hand. The first and secondarticulable portions may include fingertips having a substantiallyconvex shape. Thus, articulable portions that face toward each other,such as between a thumb articulable portion and each of the four fingerarticulable portions, may be determined to have opposing normals.

At 802, the method 800 may include determining a closest point 66 on thefirst articulable portion 58A relative to the second articulable portion58B. At 804, the method 800 may include determining a closest point 68on the second articulable portion 58B relative to the first articulableportion 58A. As illustrated with reference to FIG. 9A, the processor 12may determine the closests points 66 and 68 based on the states 48 ofthe virtual manipulator 34 and its articulable portions 58A and 58B.That is, the processor 12 may determine a point on the mesh or othercollision geometry of the first articulable portion 58A that minimizes adistance to the surface of the mesh of the second articulable portion58B, and vice versa. It should be appreciated that when the surfaces ofthe mesh of the first and second articulable portions 58A and 58B aresubstantially convex or concave, the closest points on the convex orconcave mesh/collision geometry will have opposing normals.

At 806, the method 800 may include determining a line 70 that intersectsthe closest point 66 on the first articulable portion 58A and theclosest point 68 on the second articulable portion 58B and extendsoutward from the closest points. In one example, determining the line 70may further include casting a ray in both directions along the line. Asillustrated with reference to FIG. 9A, the line 70 intersects bothclosest points 66 and 68. The line may extend outwards from the virtualmanipulator 34 for any suitable distance.

At 808, the method 800 may include detecting at least two points on thesurface of the virtual object 40 that intersect the line 70 as being theat least two contact points 56A and 56B. In the example illustrated inFIG. 9B, the processor 12 may be configured to determine that the firstarticulable portion 58A and the second articulable portion 58B arepenetrating the virtual object 40. Rather than determining the contactpoints based on an overlap between the articulable portions and thevirtual object 40, the processor 12 may be configured to identify atleast two points on the surface of the virtual object 40 that intersectthe line 70, and set those two points as the at least two contact points56A and 56B. In one example, to detect the at least two points, theprocessor 12 may be configured to cast a ray for each closest pointalong the line 70 from a negative distance, such as −10 cm. However, itshould be appreciated that any suitable distance may be used. In anotherexample, the ray may be cast backwards along the line 70 from theopposing closest point on the other articulable portion, and the contactpoint may be detected based on detecting penetration of the ray with aback face of the virtual object 40.

At 810, the method 800 may include determining the positions P1 for theat least two contact points 56A and 56B based on the intersectionsbetween the line 70 and the surface of the virtual object 40. Asillustrated with reference to FIG. 9B, the positions P1 of the contactpoints 56A and 56B are located at an interaction between the line 70,which may be cast outwards from the virtual manipulator 34, and theoutside surface of the virtual object 40, and not at a location of anintersection between a surface of the virtual manipulator 34 and thevirtual object 40. By determining contact points in this manner, themethod 800 provides the potential benefit of performing graspingcalculations that are robust to deep penetrations. That is, the method800 can calculate usable contact points for grasping and moving thevirtual object even if the virtual manipulator 34 is moved to deeplypenetrate the surfaces of the virtual object 40.

At 812, the method 800 may include determining the normal 60 for the atleast two contact points 56A and 56B to be parallel to the line 70. Asillustrated with reference to FIG. 9B, rather than calculating thenormal 60 for the at least two contact points based on a geometry of thesurface of the virtual object 40 at the contact points 56A and 56B, theprocessor 12 may be configured to set the normals 60 to be parallel tothe line 70. These positions and normals 60 for the at least two contactpoints 56A and 56B may be transformed and/or updated in the same manneras described above in method 400 with reference to FIGS. 5A and 5B. Itshould be appreciated that the techniques described in method 800provides the potential benefit of allowing interaction between thevirtual manipulators and a concave virtual object without having toapproximate the concave virtual object by a convex decomposition.

At 814, the method may include calculating the subsequent state 52B forthe virtual object 40 that minimizes an energy or residual defined interms of the one or more normals 60 for the at least two contact points56A and 56B determined for the initial state of the system 41 and theone or more normal 60 for the at least two contact points 56A and 56Bdetermined for the subsequent state of the system 41 using theposition-based energy minimizing function 44. The subsequent state 52Bfor the virtual object 40 may be calculated using similar processes andtechniques described above in method 400 with reference to FIGS. 5A and5B. FIGS. 10A, 10B, and 11 illustrate another example for calculatingthe target position 52B for the virtual object 40 that minimizes the setof residuals defined by positional quantities for at least two contactpoints.

In one example, the processor 12 may be configured to use method 800when determining whether the user intends to grasp the virtual object 40when implementing method 600. For example, the processor 12 may beconfigured to determine that the user is intending to grasp the virtualobject 40 when a pair of articulable portions of the virtual manipulatorhaving opposing normals (e.g. Thumb portion paired with each fingerportion) are penetrating the virtual object 40 as determined by firingrays in both directions along the line 70 determined in method 800. Ifany opposing pair of articulable portions of the virtual manipulator arepenetrating the virtual object 40, the processor 12 may attach thevirtual object 40 to the reference frame of the virtual manipulator asdescribed in method 600. If, in subsequent states, none of the opposingpairs of articulable portions of the virtual manipulator are detected tobe penetrating the virtual object 40, then the processor 12 may beconfigured to detach the virtual object 40 from the virtual manipulator34.

In one example, the processor 12 may be configured to simplify thegeometry of the virtual objects 40 when performing the processes andtechniques for position based energy minimizing physics simulationsdescribed herein. For example, as illustrated in FIG. 10A, the processor12 may be configured to detect the at least two contact points 56 on thesurface of the geometry of the virtual object 40. Next, the processor 12may be configured to generate a simplified geometric representation 72of the virtual object 40 based on the positions for the at least twocontact points 56 that were determined based on the intersectionsbetween the line 70 and the surface of the virtual object 40. In theillustrated example, the processor 12 generated a sphere as thesimplified geometric representation 72 to represent the virtual object40 shown in the examples of FIGS. 5A and 5B. However, it should beappreciated that the processor 12 may be configured to generate othertypes of idealized geometry to represent the virtual objects 40. Forexample, the simplified geometric representation 72 may take the form ofa rectangular box, a cylinder, a cone, or another idealized geometricshape that has fewer indices and vertices than the mesh of the virtualobject 40. The processor 12 may be configured to perform all of thetechniques and functions described herein on the simplified geometricrepresentation 72, and then correspondingly apply any movements ortransformations calculated for the simplified geometric representation72 to the virtual object 40.

FIG. 10B illustrates a schematic view for illustrative purposes of atleast two portions 58A and 58B penetrating the surface of the simplifiedgeometric representation 72 that was generated for the virtual object40. Implementing the method 800, the processor 12 may be configured todetermine the closest points on the first articulable portion 58A andthe second articulable portion 58B, and determine the line 70 betweenthose two points. Next, the processor 12 may identify contact pointsbetween the line 70 and the surface of the virtual object 40, and maygenerate the simplified geometric representation 72 based on the contactpoints. That is, the simplified geometric representation 72 isdynamically defined by the positions of the contact points determinedvia ray casts along the line 70, rather than being approximated as partof a preprocessing step of the virtual object 40 as is typical in priorapproaches. FIG. 10A illustrates an example of a cone simplifiedgeometric representation, a cylinder simplified geometricrepresentation, and a sphere simplified geometric representation thathave each been dynamically defined by the positions of the contactpoints along the ray.

The processor 12 may be configured to determine one or more positionalquantities for the contact points, such as, for example, the position P1and the normal 60 for the at least one contact point 56 between theportion 58A and 58B of the virtual manipulator 34 and a surface of thesimplified geometric representation 72 of the virtual object 40. Thesecontact points, positions, and normals 60 may then be processed andtransformed/updated according to the steps of method 400 and method 800to calculate a subsequent state for the virtual object 40, as will bedescribed in more detail below regarding FIG. 11.

FIG. 11 continues the example illustrated at FIG. 10B. In this example,the virtual manipulator 34 has moved to a subsequent state 48B.Additionally, the first articulable portion 58A and the secondarticulable portion 58B have moved to different positions relative toeach other at the subsequent state 48B. The processor 12 may thentransform and/or update the one or more positional quantities, such asthe positions and normals of the contact points, based on the subsequentstate 48B of the virtual manipulator. FIG. 11 shows a comparison of theinitial normal 60 and the transformed normal 60T. As shown, there are aset of energies or residuals R1, R2, R3, R4, R5, and R6, that may bedefined based on differences between the one or more positionalquantities of the contact points between the initial and subsequentpositions. However, as discussed above, the set of energies or residualsmay include energies or residuals that are defined based on differencesbetween positional quantities of the subsequent state of the system 41,and are not defined based on a difference between the initial state thesubsequent state of the system 41.

The processor 12 may then calculate the subsequent state for thesimplified geometric representation 72 that minimizes the set ofresiduals R1, R2, R3, R4, R5, and R6. In some examples, due to thechange in position of the two articulable portions 58A and 58B relativeto one another, not all of the residuals may necessarily be reduced tozero. However, the set of energies and/or residuals as a whole may beminimized using the position based energy minimizing function 44 for thephysics engine 42. After calculating the subsequent state for thesimplified geometric representation 72, a corresponding transformationand/or translation may be applied to the virtual object 40.

FIG. 12 illustrates another example method 1200 for detecting contactpoints between a virtual manipulator and a virtual object. At 1202, themethod 1200 may include defining a plurality of predetermined graspingpoints 74 on the portion 58A of the virtual manipulator 34. Asillustrated with reference to FIG. 13A, the plurality of predeterminedgrasping points 74 may include any suitable number of grasping points74, such as, for example, 2, 3, 4, 5, etc. These grasping points 74 arepredetermined and fixed to the portion 58A of the virtual manipulator.It should be appreciated that in examples where the virtual manipulator34 includes a plurality of articulable portions, that each articulableportion may have separate defined and fixed pluralities of predeterminedgrasping points 74. These grasping points may, for example, be selectedfrom vertices of the mesh of the virtual manipulator 34. In anotherexample, these grasping points are defined points have positionsrelative to the virtual manipulator 34.

At 1204, the method 1200 may include detecting contact points betweenthe plurality of predetermined grasping points 74 and the surface of thevirtual object 30. In one example, the processor 12 may be configured todetect these contact points based on detecting a penetration of theplurality of predetermined grasping points 74 into or near a surface ofthe virtual object 40. In another example, the contact points may bedetected based on detecting that the plurality of predetermined graspingpoints 74 are positioned within a threshold distance of a surface of thevirtual object.

After the processor 12 has detected the contact points and recognizedthat the user is attempting the grasp the virtual object 40, the method1200 may proceed to 1206. At 1206, the method 1200 may includecalculating a coordinate system transformation 76 between the pluralityof predetermined grasping points 74 at the initial state 48A of thevirtual manipulator 34 and the plurality of grasping points 74 at thesubsequent state 48B of the virtual manipulator 34. That is, as the usercontrols the virtual manipulator 34 and changes its position, becausethe plurality of grasping points 74 are fixed to the virtualmanipulator, the relative position of the plurality of grasping points74 will also correspondingly change. As illustrated with reference toFIG. 13B, by comparing the positions of the plurality of grasping points74 at the initial and subsequent states, the processor 12 may calculatea coordinate system transformation 76 that may be applied to the graspedvirtual object 40. It should be appreciated that the coordinate systemtransformation 76 may be calculated via any suitable technique, such as,for example, the Closed-form solution of absolute orientation using unitquaternions by Berthold K. P. Horn.

At 1208, the method 1200 may include calculating the subsequent state52B of the virtual object 40 based on the calculated coordinate systemtransformation 76 and positions of the contact points. The virtualobject 40 may then be moved to the subsequent state 52B and displayed tothe user via the display 20.

In the previous examples, the computer system 10 controls one virtualmanipulator 34 based on the input data 36. However, it should beappreciated that the computer system 10 may control more than onevirtual manipulator, and that the techniques and methods describedherein are also applicable grasping a virtual object 40 with two virtualmanipulators 34.

FIG. 14 illustrates an example method for grasping a virtual object 40using at least two virtual manipulators 34. At 1402, the method 1400 mayinclude controlling at least two virtual manipulators 34 based on inputdata 36 received from the input system 18. In one example, each virtualmanipulator 34 may respectively include at least a first articulableportion 58A and a second articulable portion 58B, as illustrated in theexamples of FIGS. 7A and 7B. For example, both of the user's hands maybe tracked via images captured by the camera device 22, and two virtualmanipulators may be controlled based on the tracked hands of the user.

At 1404, the method 1400 may include detecting at least two pairs ofcontact points 78 including a first pair of contact points 78A between afirst virtual manipulator 34A and the virtual object 40 and a secondpair of contact points 78B between a second virtual manipulator 34B andthe virtual object 40. As illustrated with reference to FIG. 15A, twovirtual manipulators 34A and 34B may grasp the virtual object 40 with atleast two articulable portions. The pairs of contact points 78 for thegrasping may be detected via the techniques described in methods 400 and800.

At 1406, the method 1400 may include determining one or more positionalquantities for the initial state of the system 41 that includes at leastinitial states 48A for the at least two virtual manipulators 34, theinitial state 52A for the virtual object 40, and an initial state forthe first pair of contact points 78A and second pair of contact points78B. These positional quantities may include all of the positionalquantities discussed above.

At 1408, the method 1400 may include detecting subsequent states 48B ofthe at least two virtual manipulators 34. As illustrated with referenceto FIG. 15B, the two virtual manipulators may be controlled to move tothe respective subsequent states 48B.

At 1410, the method 1400 may include determining one or more positionalquantities for the subsequent state of the system 41. The subsequentstate of the system 41 may include the subsequent states 48B of the atleast two virtual manipulators 34. The one or more positional quantitiesfor the subsequent state of the system 41 may be determined bytransforming and/or updating one or more positional quantitiesdetermined for the initial state of the system 41, such as, for example,the positions and normals 60 for the at least two pairs of contactpoints 78A and 78B. Other positional quantities of the subsequent stateof the system 41 that are not dependent upon the initial state of thesystem 41 may also be determined. In one example, each pair of contactpoints 78A and 78B may be separately processed according to thetechniques described in method 400. Thus, each contact point of the atleast two pairs of contact points 78A and 78B will have an initialposition and normal 60, and a transformed position and normal 60 thatmay be used to define one or more energies or residuals that will beminimized by the position based energy minimizing functions.

At 1412, the method 1400 may include calculating the subsequent state52B for the virtual object 40 that minimizes the set of energies orresiduals defined in terms of the one or more positional quantitiesdetermined for the initial state of the system 41 and the one or morepositional quantities determined for the subsequent state of the system41 using the position-based energy minimizing function 44. The combinedset of energies and/or residuals for all of the contact points may beminimized as a whole using the position-based energy minimizing function44 and the techniques described herein. The virtual object 40 may thenbe moved to the subsequent state 52B and displayed to the user via thedisplay 20.

In one example, the two virtual manipulators 34A and 34B may becontrolled to be moved a distance away from each other. FIG. 16illustrates an example method 1600 for resizing the virtual object 40based on the changing states of the two virtual manipulators 34A and34B. In these examples, the one or more positional quantities mayinclude scaling frames of reference that are determined for each pair ofcontact points. At 1602, the method 1600 may include determining a firstscaling frame of reference 80A based on the first pair of contact points78A and a second scaling frame of reference 80B based on the second pairof contact points 78B. As illustrated with reference to FIG. 17A, theprocessor 12 may be configured to set a size, position, and orientationof the first and second scaling frames of reference 80A and 80B based onthe respective pairs of contact points 78A and 78B. In the illustratedexample, the first and second scaling frames of reference 80A and 80Bare spheres that have been positioned between the pairs of contactpoints. The scaling frames of reference may be generated and positionedusing the similar logic as used to generate the simplified geometricrepresentations of FIG. 10A.

At 1604, the method 1600 may include calculating an initial scalingdistance 82A between the first scaling frame of reference 80A and thesecond scaling frame of reference 80B for the initial state of thesystem 41. As illustrated with reference to FIG. 17A, the initialscaling distance 82A may be calculated by determining a line between thefirst and second scaling frames of reference 80A and 80B. In oneexample, the processor 12 may be configured to generate a geometricshape, such as a cylinder, having a length dimension that spans betweenthe pairs of contact points, and the initial scaling distance 82A may becalculated based on the length of the geometric shape.

At 1606, the method 1600 may include calculating a subsequent scalingdistance 82B between the first scaling frame of reference 80A and thesecond scaling frame of reference 80B based on the subsequent states 48Bof the at least two virtual manipulators 34A and 34B for the subsequentstate of the system 41. As illustrated with reference to FIG. 17B, thetwo virtual manipulators 34A and 34B have moved to the subsequent states48B. In this example, the first virtual manipulator 34A is farther awayfrom the second virtual manipulator 34B compared to the initial states48A. The processor 12 may then transform the scaling frames of reference80A and 80B based on the subsequent states 48B of the virtualmanipulators 34A and 34B. As illustrated, the processor 12 may thencalculate the subsequent scaling distance 82B that measures a distancebetween the locations of the scaling frames of reference 80A and 80B forthe subsequent state of the system 41.

At 1608, the method 1600 may include scaling a size of the virtualobject 40 based on a ratio between the initial scaling distance 82A andthe subsequent scaling distance 82B. As illustrated with reference toFIG. 17B, the virtual object 40 has been uniformly scaled proportionallyto the ratio between the initial and subsequent scaling distances. Inanother example, the virtual object 40 may be stretched rather thanuniformly scaled.

Using the techniques described in methods 400, 600, 800, 1200, 1400, and1600, the computer system 10 may manipulate the positions, orientation,and sizes of virtual objects 40 with one or more virtual manipulators 34using position based energy minimizing functions 44. It should beappreciated that while the drawings and the foregoing descriptioninclude examples of one or two virtual manipulators interacting with asingle object floating in three-dimensional space of the virtualenvironment, the position based energy minimizing functions andprocesses described herein may be configured to process systems thatinclude any number of virtual manipulators and virtual objects. Forexample, systems that include three or more virtual manipulatorsinteracting with a plurality of virtual objects may also be processedvia energy minimizing according to the methods described herein.Additionally, it should be appreciated that the described position basedenergy minimization may also be performed on virtual objects that areconstrained in one or more degrees (e.g. a lever or button attached to aworld-locked virtual object or frame, a doorknob constrained to rotatearound an axis, etc.).

With reference to FIG. 11, mathematics related to the rules andassumptions used in an example position-based energy minimizing function44 that may be utilized by the above described systems and methods willnow be described. In one example, energy minimization may be solvedusing a Levenberg-Marquardt algorithm, as will be shown below. Usingsuch a solver, the processor 12 may be configured to perform the energyminimizing functions and processes described herein. However, it shouldbe appreciated that other suitable general-purpose algorithms or customalgorithms may be used to simultaneously minimize quadratic forms orother energy functions.

Energy-Minimizing Position Based Dynamics Equations of MotionEquivalence Of Implicit Time Integration and Energy MinimizationDefinitions

-   -   (x_(t), v_(t), a_(t))≡system state at time t (linear and        angular)    -   f_(t) ^(i) ≡force and torque (wrench) i at time t    -   M≡mass matrix    -   h≡Δt

Forward Predicted Future State

-   -   Using quantities at the current time    -   Momentum and constant external forces are applied    -   Constant external force may typically be gravity    -   Predict substantially no movement when a virtual object is        grasped    -   Apply damping    -   Vt is the difference between current and previous positions    -   Twice as much as acceleration as the formula s=ut+½at²

s _(t+1) ≡x _(t) +hv _(t) h ² M ⁻¹ f ^(constant)

Solver Prediction Correction

Δx≡x _(t+1) −s _(t+1)

Positionally-Defined Momentum Potential (“Kinetic Energy”)

E _(t+1) ^(kinetic)≡½Δx ^(T) MΔx

MΔx=∇E _(t+1) ^(kinetic)

Definition of Newtonian Implicit Time Integration

x _(t+1) =x _(t) hv _(t+1)  (1)

v _(t+1) =v _(t) +ha _(t+1)  (2)

f ^(constant)+Σ_(i) f _(t+1) ^(i) =Ma _(t+1)  (3)

Position-Dependent Conservative Forces Assumption

-   -   Future forces are a function of future positions    -   Forces can be written as the gradient of a scalar potential        energy function    -   Forces are not a function of velocities

f _(t+1) ^(i)(x _(t+1))=−∇E _(t+1) ^(i)(x _(t+1))  (4)

Equivalence Derivation

x _(t+1) =x _(t) +hv _(t) +h ² M ⁻¹[f ^(constant)+Σ_(i) f _(t+1)^(i)]  (5)

x _(t+1) =s _(t+1) +h ² M ⁻¹Σ_(i) f _(t+1) ^(i)  (6)

MΔx=h ²Σ_(i) f _(t+1) ^(i)  (7)

MΔx=−h ²Σ_(i) ∇E _(t+1) ^(i)(x _(t+1))  (8)

0=∇E _(t+1) ^(kinetic)(x _(t+1))+h ²Σ_(i) ∇E _(t+1) ^(i)(x _(t+1))  (9)

Solve (9) by Energy Minimization

-   -   Solution to (9) is also the solution to (10)    -   Typical energy terms are quadratic

$\begin{matrix}{x_{t + 1} = {\arg \mspace{11mu} {\min\limits_{x}\lbrack {{E_{t + 1}^{kinetic}(x)} + {h^{2}{\sum_{i}{E_{t + 1}^{i}(x)}}}} \rbrack}}} & (10)\end{matrix}$

Energies

Each of these terms have tune able associated weight K(“stiffness”).

Kinetic Energy

Ignore this term to solve for equilibrium as if all spring-likeoscillations have ceased. This term is unnecessary to have a uniquesolution in the example of two antipodal 6 DOF contact points.

This term enables forward dynamics such as after a virtual object isreleased from a grasp.

-   -   v≡Δpos_component_of_x    -   ω≡Δangular_component_of_x

E ^(kinetic)=½mv ²+½Iω ²

Angular Friction (Torsional, Rolling)/Thermal Energy

For each contact point k, minimize the angle between its new frame andits object-locked frame at the hypothetical new body pose, minimizerolling and torsional residuals independently to weight themdifferently, and/or minimize the cosine distance between the angles.

E ^(k,angfric)=½theta_residual_(k) ²

Elastic Potential Energy

For each contact point k, calculate a depth measured along an outersphere's line segment at the body's hypothetical new pose. Because twoof these terms exist in opposition, they will center the virtual objectbetween the fingertips of the virtual manipulator. In one example, oneterm per body be used to center the object along a skewer axis. Thisexample may provide potential benefits when a penetration is large bycausing the optimal energy to be more likely to be close to zero. AHooke's law spring has the quadratic form:

E ^(k,elastic)=½normal_pen_depth_(k) ²

Linear Friction/Thermal Energy

For each contact point k, minimize perpendicular distance from thebody's hypothetically posed line segment.

E ^(k,linfric)=½perp_residual_(k) ²

E ^(k,linfric)=½perp_residual_(k) ²

Jacobians Parameterizations and Notation

6 DOF (position, axis * angle) parameterization θ_(b) ¹ for the body (tobe solved for).

6 DOF (position, axis * angle) parameterization θ_(r,k) ¹ for eachcontact point's residual (to be minimized)

Intermediate conversion to quaternions or affine transform forcalculating quantities and their derivatives.

Notation for object a:

-   -   s_(a)≡linear position∈        ³    -   v_(a)≡angle times axis orientation∈        ³    -   q_(a)≡quaternion orientation∈S³    -   θ_(a)≡(s_(a), v_(a))    -   ϕ(s_(a), q_(a))    -   M_(a)≡rotation matrix∈        (3)    -   A≡homogeneous transform∈        (3)    -   superscript∈{0,1}≡ at time 0 or 1

Calculating Per-Contact Residual

Each contact point provides a 6-component residual vector, which iscalculated by multiplying affine transforms or quaternions, andconverting the resulting transform into a (position, axis * angle)vector.

The transform from which we can read residual positions and angles:

R _(k)=(L _(k))⁻¹·(B ¹)⁻¹ ·F _(k) ¹  (1)

Which can be regarded as having this mathematical form:

R _(k) =A _(k) ·X(s _(x) ,q _(x)(v _(x)))·C _(k)  (2)

q _(r,k) =q _(a,k) ·q _(x) ·q _(c,k)  (3)

Where A_(k) and C_(k) are constant known transforms.

Residuals and Jacobians

$\begin{matrix}\theta_{r,k} & (4) \\\frac{\partial\theta_{r,k}}{\partial\theta_{x}} & (5)\end{matrix}$

Log Map Jacobian

$\begin{matrix}\frac{\partial v_{r,k}}{\partial q_{r,k}} & (6)\end{matrix}$

Exponential Map Jacobian

$\begin{matrix}\frac{\partial q_{x}}{\partial v_{x}} & (7)\end{matrix}$

Derivative of Rotation Matrix wrt Quarternion

$\begin{matrix}\frac{\partial M_{x}}{\partial q_{x}} & (8)\end{matrix}$

Derivatives of Angular Residuals with Respect to (Inverse) BodyOrientation:(3), (5), (6)=>

$\begin{matrix}\frac{\partial v_{r,k}}{\partial v_{x}} & (8)\end{matrix}$

Derivatives of Linear Residuals with Respect to (Inverse) Body Positionand Orientation(2)=>

$\begin{matrix}\frac{\partial s_{r,k}}{\partial s_{x}} & (9)\end{matrix}$

(2), (6), (7)=>

$\begin{matrix}\frac{\partial s_{r,k}}{\partial v_{x}} & (10)\end{matrix}$

Derivatives of Angular Residuals with Respect to (Inverse) Body Position

$\begin{matrix}\frac{\partial v_{r,k}}{\partial s_{x}} & (11)\end{matrix}$

In some embodiments, the methods and processes described herein may betied to a computing system of one or more computing devices. Inparticular, such methods and processes may be implemented as acomputer-application program or service, an application-programminginterface (API), a library, and/or other computer-program product.

FIG. 18 schematically shows a non-limiting embodiment of a computingsystem 1800 that can enact one or more of the methods and processesdescribed above. Computing system 1800 is shown in simplified form.Computing system 1800 may embody the computer system 10 described aboveand illustrated in FIG. 1. Computing system 1800 may take the form ofone or more personal computers, server computers, tablet computers,home-entertainment computers, network computing devices, gaming devices,mobile computing devices, mobile communication devices (e.g., smartphone), and/or other computing devices, and wearable computing devicessuch as smart wristwatches and head mounted augmented reality devices.

Computing system 1800 includes a logic processor 1802 volatile memory1804, and a non-volatile storage device 1806. Computing system 1800 mayoptionally include a display subsystem 1808, input subsystem 1810,communication subsystem 1812, and/or other components not shown in FIG.18.

Logic processor 1802 includes one or more physical devices configured toexecute instructions. For example, the logic processor may be configuredto execute instructions that are part of one or more applications,programs, routines, libraries, objects, components, data structures, orother logical constructs. Such instructions may be implemented toperform a task, implement a data type, transform the state of one ormore components, achieve a technical effect, or otherwise arrive at adesired result.

The logic processor may include one or more physical processors(hardware) configured to execute software instructions. Additionally oralternatively, the logic processor may include one or more hardwarelogic circuits or firmware devices configured to executehardware-implemented logic or firmware instructions. Processors of thelogic processor 1802 may be single-core or multi-core, and theinstructions executed thereon may be configured for sequential,parallel, and/or distributed processing. Individual components of thelogic processor optionally may be distributed among two or more separatedevices, which may be remotely located and/or configured for coordinatedprocessing. Aspects of the logic processor may be virtualized andexecuted by remotely accessible, networked computing devices configuredin a cloud-computing configuration. In such a case, these virtualizedaspects are run on different physical logic processors of variousdifferent machines, it will be understood.

Non-volatile storage device 1806 includes one or more physical devicesconfigured to hold instructions executable by the logic processors toimplement the methods and processes described herein. When such methodsand processes are implemented, the state of non-volatile storage device1806 may be transformed—e.g., to hold different data.

Non-volatile storage device 1806 may include physical devices that areremovable and/or built-in. Non-volatile storage device 1806 may includeoptical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.),semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.),and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tapedrive, MRAM, etc.), or other mass storage device technology.Non-volatile storage device 1806 may include nonvolatile, dynamic,static, read/write, read-only, sequential-access, location-addressable,file-addressable, and/or content-addressable devices. It will beappreciated that non-volatile storage device 1806 is configured to holdinstructions even when power is cut to the non-volatile storage device1806.

Volatile memory 1804 may include physical devices that include randomaccess memory. Volatile memory 1804 is typically utilized by logicprocessor 1802 to temporarily store information during processing ofsoftware instructions. It will be appreciated that volatile memory 1804typically does not continue to store instructions when power is cut tothe volatile memory 1804.

Aspects of logic processor 1802, volatile memory 1804, and non-volatilestorage device 1806 may be integrated together into one or morehardware-logic components. Such hardware-logic components may includefield-programmable gate arrays (FPGAs), program- andapplication-specific integrated circuits (PASIC/ASICs), program- andapplication-specific standard products (PSSP/ASSPs), system-on-a-chip(SOC), and complex programmable logic devices (CPLDs), for example.

The terms “module,” “program,” and “engine” may be used to describe anaspect of computing system 1800 typically implemented in software by aprocessor to perform a particular function using portions of volatilememory, which function involves transformative processing that speciallyconfigures the processor to perform the function. Thus, a module,program, or engine may be instantiated via logic processor 1802executing instructions held by non-volatile storage device 1806, usingportions of volatile memory 1804. It will be understood that differentmodules, programs, and/or engines may be instantiated from the sameapplication, service, code block, object, library, routine, API,function, etc. Likewise, the same module, program, and/or engine may beinstantiated by different applications, services, code blocks, objects,routines, APIs, functions, etc. The terms “module,” “program,” and“engine” may encompass individual or groups of executable files, datafiles, libraries, drivers, scripts, database records, etc.

When included, display subsystem 1808 may be used to present a visualrepresentation of data held by non-volatile storage device 1806. Thevisual representation may take the form of a graphical user interface(GUI). As the herein described methods and processes change the dataheld by the non-volatile storage device, and thus transform the state ofthe non-volatile storage device, the state of display subsystem 1808 maylikewise be transformed to visually represent changes in the underlyingdata. Display subsystem 1808 may include one or more display devicesutilizing virtually any type of technology. Such display devices may becombined with logic processor 1802, volatile memory 1804, and/ornon-volatile storage device 1806 in a shared enclosure, or such displaydevices may be peripheral display devices.

When included, input subsystem 1810 may comprise or interface with oneor more user-input devices such as a keyboard, mouse, touch screen, orgame controller. In some embodiments, the input subsystem may compriseor interface with selected natural user input (NUI) componentry. Suchcomponentry may be integrated or peripheral, and the transduction and/orprocessing of input actions may be handled on- or off-board. Example NUIcomponentry may include a microphone for speech and/or voicerecognition; an infrared, color, stereoscopic, and/or depth camera formachine vision and/or gesture recognition; a head tracker, eye tracker,accelerometer, and/or gyroscope for motion detection and/or intentrecognition; as well as electric-field sensing componentry for assessingbrain activity; and/or any other suitable sensor.

When included, communication subsystem 1812 may be configured tocommunicatively couple various computing devices described herein witheach other, and with other devices. Communication subsystem 1812 mayinclude wired and/or wireless communication devices compatible with oneor more different communication protocols. As non-limiting examples, thecommunication subsystem may be configured for communication via awireless telephone network, or a wired or wireless local- or wide-areanetwork, such as a HDMI over Wi-Fi connection. In some embodiments, thecommunication subsystem may allow computing system 1800 to send and/orreceive messages to and/or from other devices via a network such as theInternet.

The following paragraphs provide additional support for the claims ofthe subject application. One aspect provides a computer systemcomprising an input system, and a processor configured to control avirtual manipulator based on input data received from the input system,the virtual manipulator being configured to manipulate a virtual objectin a virtual environment. The processor is further configured todetermine an initial state of a system that includes at least an initialstate of the virtual manipulator and an initial state of the virtualobject, detect at least one contact point between a portion of thevirtual manipulator and the virtual object, determine one or morepositional quantities for the initial state of the system, detect asubsequent state of the virtual manipulator, and determine one or morepositional quantities for a subsequent state of the system that includesthe subsequent state of the virtual manipulator. The processor isfurther configured to calculate a subsequent state for the virtualobject that minimizes a set of energies or residuals defined in terms ofthe one or more positional quantities determined for the initial stateof the system and the one or more positional quantities determined forthe subsequent state of the system using a position-based energyminimizing function. The processor is further configured to move thevirtual object to the calculated subsequent state. In this aspect,additionally or alternatively, the one or more positional quantities maybe selected from the group consisting of a frame of reference, a contactpoint tangential component for friction, a position, a normal, and apenetration depth. In this aspect, additionally or alternatively, theset of energies or residuals may be selected from the group consistingof a kinetic energy, a penetration depth residual, a tangential frictionresidual, an angular friction residual, a normal residual, and a frameof reference residual. In this aspect, additionally or alternatively,the initial state and the subsequent state of the virtual object mayinclude three-dimensional positions and orientations of the virtualobject in the virtual environment, and wherein the initial state and thesubsequent state of the virtual manipulator may include respectivethree-dimensional positions and orientations of the portion of thevirtual manipulator in the virtual environment. In this aspect,additionally or alternatively, the processor may be configured to tracka position of a user's hand via the input data received from the inputsystem, and the virtual manipulator may be a graphical representation ofa hand or grasping tool that is controlled based on the tracked positionof the user's hand. In this aspect, additionally or alternatively, theone or more positional quantities may include a frame of reference ofthe virtual manipulator, and the processor may be configured to, basedon detecting the at least one contact point between the portion of thevirtual manipulator and the virtual object, determine a relative stateof the virtual object compared to the frame of reference of the virtualmanipulator. The processor may be further configured to calculate thesubsequent state for the virtual object that minimizes an energy orresidual defined in terms of the relative state of the virtual objectcompared to the frame of reference of the virtual manipulator for theinitial state of the system and the relative state of the virtual objectcompared to the frame of reference of the virtual manipulator for thesubsequent state of the system. In this aspect, additionally oralternatively, the portion of the virtual manipulator may include atleast a first articulable portion and a second articulable portion, andthe processor may be further configured to detect at least two contactpoints including a first contact point between the first articulableportion and the virtual object and a second contact point between thesecond articulable portion and the virtual object. In this aspect,additionally or alternatively, to detect the at least two contactpoints, the processor may be further configured to determine a closestpoint on the first articulable portion relative to the secondarticulable portion, determine a closest point on the second articulableportion relative to the first articulable portion, determine a line thatintersects the closest point on the first articulable portion and theclosest point on the second articulable portion and extends outward fromthe closest points, detect at least two points on the surface of thevirtual object that intersect the line as being the at least two contactpoints, and determine the positions for the at least two contact pointsbased on the intersections between the line and the surface of thevirtual object. In this aspect, additionally or alternatively, theprocessor may be further configured to generate a simplified geometricrepresentation of the virtual object based on the positions for the atleast two contact points that were determined based on the intersectionsbetween the line and the surface of the virtual object. In this aspect,additionally or alternatively, the one or more positional quantities mayinclude a normal, and the processor may be further configured todetermine one or more normals for the at least two contact points to beparallel to the line, and the processor may be configured to calculatethe subsequent state for the virtual object that minimizes an energy orresidual defined in terms of the one or more normals for the at leasttwo contact points determined for the initial state of the system andthe one or more normals for the at least two contact points determinedfor the subsequent state of the system using the position-based energyminimizing function. In this aspect, additionally or alternatively, theprocessor may be configured to control at least two virtual manipulatorsbased on input data received from the input system, each virtualmanipulator respectively including at least a first articulable portionand a second articulable portion. The processor may be furtherconfigured to detect at least two pairs of contact points including afirst pair of contact points between a first virtual manipulator and thevirtual object and a second pair of contact points between a secondvirtual manipulator and the virtual object, and determine one or morepositional quantities for the initial state of the system that includesat least initial states for the at least two virtual manipulators, theinitial state for the virtual object, and initial states for the firstpair of contact points and second pair of contact points. The processormay be further configured to detect subsequent states of the at leasttwo virtual manipulators, determine one or more positional quantitiesfor the subsequent state of the system, and calculate the subsequentstate for the virtual object that minimizes the set of energies orresiduals defined in terms of the one or more positional quantitiesdetermined for the initial state of the system and the one or morepositional quantities determined for the subsequent state of the systemusing the position-based energy minimizing function. In this aspect,additionally or alternatively, the one or more positional quantities mayinclude a scaling frame of reference, and the processor may beconfigured to determine a first scaling frame of reference based on thefirst pair of contact points and a second scaling frame of referencebased on the second pair of contact points, calculate an initial scalingdistance between the first scaling frame of reference and the secondscaling frame reference for the initial state of the system, calculate asubsequent scaling distance between the first scaling frame of referenceand the second scaling frame of reference based on the subsequent statesof the at least two virtual manipulators for the subsequent state of thesystem, and scale a size of the virtual object based on a ratio betweenthe initial scaling distance and the subsequent scaling distance. Inthis aspect, additionally or alternatively, the processor may beconfigured to define a plurality of predetermined grasping points on theportion of the virtual manipulator, detect contact points between theplurality of predetermined grasping points and the surface of thevirtual object, and calculate a coordinate system transformation betweenthe plurality of predetermined grasping points at the initial state ofthe virtual manipulator and the plurality of grasping points at thesubsequent state of the virtual manipulator. The processor may befurther configured to calculate the subsequent state of the virtualobject based on the calculated coordinate system transformation andpositions of the contact points.

Another aspect provides a method comprising, at a processor, controllinga virtual manipulator based on input data received from an input system,the virtual manipulator being configured to manipulate a virtual objectin a virtual environment. The method may further include determining aninitial state of a system that includes at least an initial state of thevirtual manipulator and an initial state of the virtual object,detecting at least one contact point between a portion of the virtualmanipulator and the virtual object, determining one or more positionalquantities for the initial state of the system, detecting a subsequentstate of the virtual manipulator, and determining one or more positionalquantities for a subsequent state of the system that includes thesubsequent state of the virtual manipulator. The method may furtherinclude calculating a subsequent state for the virtual object thatminimizes a set of energies or residuals defined in terms of the one ormore positional quantities determined for the initial state of thesystem and the one or more positional quantities determined for thesubsequent state of the system using a position-based energy minimizingfunction. The method may further include moving the virtual object tothe calculated subsequent state. In this aspect, additionally oralternatively, the one or more positional quantities may include a frameof reference of the virtual manipulator, and the method may furtherinclude, based on detecting the at least one contact point between theportion of the virtual manipulator and the virtual object, determining arelative state of the virtual object compared to the frame of referenceof the virtual manipulator. The method may further include calculatingthe subsequent state for the virtual object that minimizes an energy orresidual defined in terms of the relative state of the virtual objectcompared to the frame of reference of the virtual manipulator for theinitial state of the system and the relative state of the virtual objectcompared to the frame of reference of the virtual manipulator for thesubsequent state of the system. In this aspect, additionally oralternatively, the portion of the virtual manipulator may include atleast a first articulable portion and a second articulable portion, andthe method may further include determining a closest point on the firstarticulable portion relative to the second articulable portion,determining a closest point on the second articulable portion relativeto the first articulable portion, determining a line that intersects theclosest point on the first articulable portion and the closest point onthe second articulable portion and extends outward from the closestpoints, detecting at least two points on the surface of the virtualobject that intersect the line as being at least two contact points, anddetermining the position for the at least two contact points based onthe intersections between the line and the surface of the virtualobject. In this aspect, additionally or alternatively, the one or morepositional quantities may include a normal, and the method may furtherinclude determining one or more normals for the at least two contactpoints to be parallel to the line. The method may further includecalculating the subsequent state for the virtual object that minimizesan energy or residual defined in terms of the one or more normals forthe at least two contact points determined for the initial state of thesystem and the one or more normals for the at least two contact pointsdetermined for the subsequent state of the system using theposition-based energy minimizing function. In this aspect, additionallyor alternatively, the method may further include controlling at leasttwo virtual manipulators based on input data received from the inputsystem, each virtual manipulator respectively including at least a firstarticulable portion and a second articulable portion. The method mayfurther include detecting at least two pairs of contact points includinga first pair of contact points between a first virtual manipulator andthe virtual object and a second pair of contact points between a secondvirtual manipulator and the virtual object, and determining one or morepositional quantities for the initial state of the system that includesat least initial states for the at least two virtual manipulators, theinitial state for the virtual object, and initial states for the firstpair of contact points and second pair of contact points. The method mayfurther include detecting subsequent states of the at least two virtualmanipulators, determining one or more positional quantities for thesubsequent state of the system, and calculating the subsequent state forthe virtual object that minimizes the set of energies or residualsdefined in terms of the one or more positional quantities determined forthe initial state of the system and the one or more positionalquantities determined for the subsequent state of the system using theposition-based energy minimizing function. In this aspect, additionallyor alternatively, the one or more positional quantities may include ascaling frame of reference, and the method may further includedetermining a first scaling frame of reference based on the first pairof contact points and a second scaling frame of reference based on thesecond pair of contact points, calculating an initial scaling distancebetween the first scaling frame of reference and the second scalingframe of reference for the initial state of the system, calculating asubsequent scaling distance between the first scaling frame of referenceand the second scaling frame of reference based on the subsequentpositions of the at least two virtual manipulators for the subsequentstate of the system, and scaling a size of the virtual object based on aratio between the initial scaling distance and the subsequent scalingdistance.

Another aspect provides a head computed display device comprising anear-eye display, a camera system configured to capture images of auser's hand, and a processor. The processor is configured to track aposition of the user's hand based on the captured images received fromthe camera system, and control a virtual manipulator based on thetracked position of the user's hand, the virtual manipulator beingconfigured to manipulate a virtual object in a virtual environment. Theprocessor is further configured to determine an initial state of asystem that includes at least an initial state of the virtualmanipulator and an initial state of the virtual object, detect at leastone contact point between a portion of the virtual manipulator and thevirtual object, determine one or more positional quantities for theinitial state of the system, detect a subsequent state of the virtualmanipulator, and determine one or more positional quantities for asubsequent state of the system that includes the subsequent state of thevirtual manipulator. The processor is further configured to calculate asubsequent state for the virtual object that minimizes a set of energiesor residuals defined in terms of the one or more positional quantitiesdetermined for the initial state of the system and the one or morepositional quantities determined for the subsequent state of the systemusing a position-based energy minimizing function. The processor isfurther configured to move the virtual object to the calculatedsubsequent state and display the subsequent state of the system in thevirtual environment via the near-eye display.

It will be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated and/ordescribed may be performed in the sequence illustrated and/or described,in other sequences, in parallel, or omitted. Likewise, the order of theabove-described processes may be changed.

The subject matter of the present disclosure includes all novel andnon-obvious combinations and sub-combinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

1. A computer system comprising: an input system; and a processorconfigured to: control a virtual manipulator based on input datareceived from the input system, the virtual manipulator being configuredto manipulate a virtual object in a virtual environment; determine aninitial state of a system that includes at least an initial state of thevirtual manipulator and an initial state of the virtual object; detectat least one contact point between a portion of the virtual manipulatorand the virtual object; determine one or more positional quantities forthe initial state of the system; detect a subsequent state of thevirtual manipulator; determine one or more positional quantities for asubsequent state of the system that includes the subsequent state of thevirtual manipulator; calculate a subsequent state for the virtual objectthat minimizes a set of energies or residuals defined in terms of theone or more positional quantities determined for the initial state ofthe system and the one or more positional quantities determined for thesubsequent state of the system using a position-based energy minimizingfunction; and move the virtual object to the calculated subsequentstate.
 2. The computer system of claim 1, wherein the one or morepositional quantities are selected from the group consisting of a frameof reference, a contact point tangential component for friction, aposition, a normal, and a penetration depth.
 3. The computer system ofclaim 1, wherein the set of energies or residuals are selected from thegroup consisting of a kinetic energy, a penetration depth residual, atangential friction residual, an angular friction residual, a normalresidual, and a frame of reference residual.
 4. The computer system ofclaim 1, wherein the initial state and the subsequent state of thevirtual object include three-dimensional positions and orientations ofthe virtual object in the virtual environment, and wherein the initialstate and the subsequent state of the virtual manipulator includerespective three-dimensional positions and orientations of the portionof the virtual manipulator in the virtual environment.
 5. The computersystem of claim 1, wherein the processor is configured to track aposition of a user's hand via the input data received from the inputsystem; and wherein the virtual manipulator is a graphicalrepresentation of a hand or grasping tool that is controlled based onthe tracked position of the user's hand.
 6. The computer system of claim1, wherein the one or more positional quantities include a frame ofreference of the virtual manipulator, and wherein the processor isconfigured to: based on detecting the at least one contact point betweenthe portion of the virtual manipulator and the virtual object, determinea relative state of the virtual object compared to the frame ofreference of the virtual manipulator; and calculate the subsequent statefor the virtual object that minimizes an energy or residual defined interms of the relative state of the virtual object compared to the frameof reference of the virtual manipulator for the initial state of thesystem and the relative state of the virtual object compared to theframe of reference of the virtual manipulator for the subsequent stateof the system.
 7. The computer system of claim 1, wherein the portion ofthe virtual manipulator includes at least a first articulable portionand a second articulable portion; and wherein the processor is furtherconfigured to detect at least two contact points including a firstcontact point between the first articulable portion and the virtualobject and a second contact point between the second articulable portionand the virtual object.
 8. The computer system of claim 7, wherein todetect the at least two contact points, the processor is furtherconfigured to: determine a closest point on the first articulableportion relative to the second articulable portion; determine a closestpoint on the second articulable portion relative to the firstarticulable portion; determine a line that intersects the closest pointon the first articulable portion and the closest point on the secondarticulable portion and extends outward from the closest points; detectat least two points on the surface of the virtual object that intersectthe line as being the at least two contact points; and determine thepositions for the at least two contact points based on the intersectionsbetween the line and the surface of the virtual object.
 9. The computersystem of claim 8, wherein the processor is further configured to:generate a simplified geometric representation of the virtual objectbased on the positions for the at least two contact points that weredetermined based on the intersections between the line and the surfaceof the virtual object.
 10. The computer system of claim 8, wherein theone or more positional quantities include a normal, and the processor isfurther configured to determine one or more normals for the at least twocontact points to be parallel to the line; and wherein the processor isconfigured to calculate the subsequent state for the virtual object thatminimizes an energy or residual defined in terms of the one or morenormals for the at least two contact points determined for the initialstate of the system and the one or more normals for the at least twocontact points determined for the subsequent state of the system usingthe position-based energy minimizing function.
 11. The computer systemof claim 7, wherein the processor is configured to: control at least twovirtual manipulators based on input data received from the input system,each virtual manipulator respectively including at least a firstarticulable portion and a second articulable portion; detect at leasttwo pairs of contact points including a first pair of contact pointsbetween a first virtual manipulator and the virtual object and a secondpair of contact points between a second virtual manipulator and thevirtual object; determine one or more positional quantities for theinitial state of the system that includes at least initial states forthe at least two virtual manipulators, the initial state for the virtualobject, and initial states for the first pair of contact points andsecond pair of contact points; detect subsequent states of the at leasttwo virtual manipulators; determine one or more positional quantitiesfor the subsequent state of the system; calculate the subsequent statefor the virtual object that minimizes the set of energies or residualsdefined in terms of the one or more positional quantities determined forthe initial state of the system and the one or more positionalquantities determined for the subsequent state of the system using theposition-based energy minimizing function.
 12. The computer system ofclaim 11, wherein the one or more positional quantities include ascaling frame of reference, and wherein the processor is configured to:determine a first scaling frame of reference based on the first pair ofcontact points and a second scaling frame of reference based on thesecond pair of contact points; calculate an initial scaling distancebetween the first scaling frame of reference and the second scalingframe reference for the initial state of the system; calculate asubsequent scaling distance between the first scaling frame of referenceand the second scaling frame of reference based on the subsequent statesof the at least two virtual manipulators for the subsequent state of thesystem; and scale a size of the virtual object based on a ratio betweenthe initial scaling distance and the subsequent scaling distance. 13.The computer system of claim 1, wherein the processor is configured to:define a plurality of predetermined grasping points on the portion ofthe virtual manipulator; detect contact points between the plurality ofpredetermined grasping points and the surface of the virtual object;calculate a coordinate system transformation between the plurality ofpredetermined grasping points at the initial state of the virtualmanipulator and the plurality of grasping points at the subsequent stateof the virtual manipulator; and wherein the processor is furtherconfigured to calculate the subsequent state of the virtual object basedon the calculated coordinate system transformation and positions of thecontact points.
 14. A method comprising: at a processor: controlling avirtual manipulator based on input data received from an input system,the virtual manipulator being configured to manipulate a virtual objectin a virtual environment; determining an initial state of a system thatincludes at least an initial state of the virtual manipulator and aninitial state of the virtual object; detecting at least one contactpoint between a portion of the virtual manipulator and the virtualobject; determining one or more positional quantities for the initialstate of the system; detecting a subsequent state of the virtualmanipulator; determining one or more positional quantities for asubsequent state of the system that includes the subsequent state of thevirtual manipulator; calculating a subsequent state for the virtualobject that minimizes a set of energies or residuals defined in terms ofthe one or more positional quantities determined for the initial stateof the system and the one or more positional quantities determined forthe subsequent state of the system using a position-based energyminimizing function; and moving the virtual object to the calculatedsubsequent state.
 15. The method of claim 14, wherein the one or morepositional quantities include a frame of reference of the virtualmanipulator, and wherein the method further comprises: based ondetecting the at least one contact point between the portion of thevirtual manipulator and the virtual object, determining a relative stateof the virtual object compared to the frame of reference of the virtualmanipulator; and calculating the subsequent state for the virtual objectthat minimizes an energy or residual defined in terms of the relativestate of the virtual object compared to the frame of reference of thevirtual manipulator for the initial state of the system and the relativestate of the virtual object compared to the frame of reference of thevirtual manipulator for the subsequent state of the system.
 16. Themethod of claim 14, wherein the portion of the virtual manipulatorincludes at least a first articulable portion and a second articulableportion; and wherein the method further comprises: determining a closestpoint on the first articulable portion relative to the secondarticulable portion; determining a closest point on the secondarticulable portion relative to the first articulable portion;determining a line that intersects the closest point on the firstarticulable portion and the closest point on the second articulableportion and extends outward from the closest points; detecting at leasttwo points on the surface of the virtual object that intersect the lineas being at least two contact points; and determining the position forthe at least two contact points based on the intersections between theline and the surface of the virtual object.
 17. The method of claim 16,wherein the one or more positional quantities include a normal, and themethod further comprises: determining one or more normals for the atleast two contact points to be parallel to the line; and calculating thesubsequent state for the virtual object that minimizes an energy orresidual defined in terms of the one or more normals for the at leasttwo contact points determined for the initial state of the system andthe one or more normals for the at least two contact points determinedfor the subsequent state of the system using the position-based energyminimizing function.
 18. The method of claim 16, further comprising:controlling at least two virtual manipulators based on input datareceived from the input system, each virtual manipulator respectivelyincluding at least a first articulable portion and a second articulableportion; detecting at least two pairs of contact points including afirst pair of contact points between a first virtual manipulator and thevirtual object and a second pair of contact points between a secondvirtual manipulator and the virtual object; determining one or morepositional quantities for the initial state of the system that includesat least initial states for the at least two virtual manipulators, theinitial state for the virtual object, and initial states for the firstpair of contact points and second pair of contact points; detectingsubsequent states of the at least two virtual manipulators; determiningone or more positional quantities for the subsequent state of thesystem; and calculating the subsequent state for the virtual object thatminimizes the set of energies or residuals defined in terms of the oneor more positional quantities determined for the initial state of thesystem and the one or more positional quantities determined for thesubsequent state of the system using the position-based energyminimizing function.
 19. The method of claim 18, wherein the one or morepositional quantities include a scaling frame of reference, and whereinthe method further comprises: determining a first scaling frame ofreference based on the first pair of contact points and a second scalingframe of reference based on the second pair of contact points;calculating an initial scaling distance between the first scaling frameof reference and the second scaling frame of reference for the initialstate of the system; calculating a subsequent scaling distance betweenthe first scaling frame of reference and the second scaling frame ofreference based on the subsequent positions of the at least two virtualmanipulators for the subsequent state of the system; and scaling a sizeof the virtual object based on a ratio between the initial scalingdistance and the subsequent scaling distance.
 20. A head computeddisplay device comprising: a near-eye display; a camera systemconfigured to capture images of a user's hand; and a processorconfigured to: track a position of the user's hand based on the capturedimages received from the camera system; control a virtual manipulatorbased on the tracked position of the user's hand, the virtualmanipulator being configured to manipulate a virtual object in a virtualenvironment; determine an initial state of a system that includes atleast an initial state of the virtual manipulator and an initial stateof the virtual object; detect at least one contact point between aportion of the virtual manipulator and the virtual object; determine oneor more positional quantities for the initial state of the system;detect a subsequent state of the virtual manipulator; determine one ormore positional quantities for a subsequent state of the system thatincludes the subsequent state of the virtual manipulator; calculate asubsequent state for the virtual object that minimizes a set of energiesor residuals defined in terms of the one or more positional quantitiesdetermined for the initial state of the system and the one or morepositional quantities determined for the subsequent state of the systemusing a position-based energy minimizing function; move the virtualobject to the calculated subsequent state and display the subsequentstate of the system in the virtual environment via the near-eye display.